Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e: Collect git action data in case of a failure #57

Merged
merged 1 commit into from
Aug 19, 2024
Merged

e2e: Collect git action data in case of a failure #57

merged 1 commit into from
Aug 19, 2024

Conversation

oshoval
Copy link
Collaborator

@oshoval oshoval commented Aug 13, 2024 •
edited
Loading

What this PR does / why we need it:
Collect k8s logs upon errors to ease debugging.
For more info see https://github.com/actions/upload-artifact

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:
It assume there are no parallel runs of the tests (it cleans the logs folder).

Each error would consume around 0.5Mb zipped.
We might want to consider reducing the retention, so it wont limit us
but on the other hand this repo isn't super active / flaky, so we can just be proactive.

Tested on CI by injecting an error.

Upon needs we can add logs (i.e virt-launcher), describe VMIs and so on.

Release note:

@kubevirt-bot kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Aug 13, 2024
@kubevirt-bot kubevirt-bot requested review from maiqueb and qinqon August 13, 2024 09:32
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 13, 2024 •
edited
Loading

looks like this (all on stdout, if we want a file(s) it can be done as well, but git actions will need to collect it)
(i prefer doing that on a follow-up please according needs)

Persistent IPs when network attachment definition created with allowPersistentIPs=true and a virtual machine using it is also created should keep ips after live migration
/root/project/ipam-extensions/test/e2e/persistentips_test.go:102
  STEP: Creating namespace testns-2f4769f41 @ 08/13/24 12:29:42.016
  STEP: Create NetworkAttachmentDefinition @ 08/13/24 12:29:42.035
  STEP: Creating VM with secondary attachments @ 08/13/24 12:29:42.056
  STEP: Waiting for readiness at virtual machine alpine-2a94f08aa @ 08/13/24 12:29:42.08
  [FAILED] in [BeforeEach] - /root/project/ipam-extensions/test/e2e/persistentips_test.go:88 @ 08/13/24 12:30:29.505
  STEP: Test failed, collecting logs and artifacts @ 08/13/24 12:30:29.505
kubectl get pods:
NAMESPACE                         NAME                                                READY   STATUS    RESTARTS   AGE
cert-manager                      cert-manager-5d864474d-7ngct                        1/1     Running   0          10m
cert-manager                      cert-manager-cainjector-85d499d4cc-qnzqt            1/1     Running   0          10m
cert-manager                      cert-manager-webhook-d499d9459-2ddls                1/1     Running   0          10m
kube-system                       coredns-7db6d8ff4d-9nzhl                            1/1     Running   0          18m
kube-system                       coredns-7db6d8ff4d-r8n4m                            1/1     Running   0          18m
kube-system                       etcd-virt-ipam-control-plane                        1/1     Running   0          19m
kube-system                       kube-apiserver-virt-ipam-control-plane              1/1     Running   0          19m
kube-system                       kube-controller-manager-virt-ipam-control-plane     1/1     Running   0          19m
kube-system                       kube-multus-ds-6m4pw                                1/1     Running   0          16m
kube-system                       kube-multus-ds-c4gr8                                1/1     Running   0          16m
kube-system                       kube-multus-ds-wxvm7                                1/1     Running   0          16m
kube-system                       kube-scheduler-virt-ipam-control-plane              1/1     Running   0          19m
kubevirt-ipam-controller-system   kubevirt-ipam-controller-manager-66bb5f9b96-5kzzg   1/1     Running   0          10m
kubevirt                          virt-api-7976d99767-czdff                           1/1     Running   0          12m
kubevirt                          virt-api-7976d99767-qdpp2                           1/1     Running   0          12m
kubevirt                          virt-controller-bd5f9c85b-mh7zx                     1/1     Running   0          12m
kubevirt                          virt-controller-bd5f9c85b-p8579                     1/1     Running   0          12m
kubevirt                          virt-handler-c5qtm                                  1/1     Running   0          12m
kubevirt                          virt-handler-mghpq                                  1/1     Running   0          12m
kubevirt                          virt-handler-v2qj2                                  1/1     Running   0          12m
kubevirt                          virt-operator-7c7b6c4d87-dvr9w                      1/1     Running   0          13m
kubevirt                          virt-operator-7c7b6c4d87-jmvmv                      1/1     Running   0          13m
local-path-storage                local-path-provisioner-65749bdd48-tmv2n             1/1     Running   0          18m
ovn-kubernetes                    ovnkube-control-plane-6dd6df59c9-zlrsj              1/1     Running   0          16m
ovn-kubernetes                    ovnkube-identity-5859d96cbc-c5w46                   1/1     Running   0          16m
ovn-kubernetes                    ovnkube-node-2qxs6                                  6/6     Running   0          16m
ovn-kubernetes                    ovnkube-node-chdzc                                  6/6     Running   0          16m
ovn-kubernetes                    ovnkube-node-mpfns                                  6/6     Running   0          16m
ovn-kubernetes                    ovs-node-7g4zh                                      1/1     Running   0          16m
ovn-kubernetes                    ovs-node-9wv6c                                      1/1     Running   0          16m
ovn-kubernetes                    ovs-node-bghhg                                      1/1     Running   0          16m
testns-2f4769f41                  virt-launcher-alpine-2a94f08aa-ngbpt                2/2     Running   0          47s

error:
<nil>
  STEP: Deleting namespace testns-2f4769f41 @ 08/13/24 12:30:29.637
• [FAILED] [83.733 seconds]
Persistent IPs when network attachment definition created with allowPersistentIPs=true and a virtual machine using it is also created [BeforeEach] should keep ips after live migration
  [BeforeEach] /root/project/ipam-extensions/test/e2e/persistentips_test.go:77
  [It] /root/project/ipam-extensions/test/e2e/persistentips_test.go:102

  [FAILED] Expected
      <int>: 1
  to equal
      <int>: 2
  In [BeforeEach] at: /root/project/ipam-extensions/test/e2e/persistentips_test.go:88 @ 08/13/24 12:30:29.505

@oshoval oshoval mentioned this pull request Aug 13, 2024
Closed
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 13, 2024 •
edited
Loading

Btw first test flakes happens here as well
https://github.com/kubevirt/ipam-extensions/actions/runs/10367263790/job/28698299367?pr=57
this change was meant to help investigating it
It might be even a bug with OVN itself (we always take latest OVN main HEAD, but lets investigate first)

Opened an issue on CNAO, but it affects this repo as standalone as well
kubevirt/cluster-network-addons-operator#1857

@oshoval oshoval changed the title e2e: Collect data in case of a failure WIP e2e: Collect data in case of a failure Aug 13, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 13, 2024

injected an error and trying to collect to git actions artifacts folder
it will be nicer to have each log on its own file instead of stdout

@oshoval oshoval force-pushed the collect branch 4 times, most recently from 8d9bbe9 to 716a003 Compare August 13, 2024 13:41
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 13, 2024 •
edited
Loading

The files are uploaded in the job,
will rework it

Run actions/upload-artifact@v4
With the provided path, there will be 6 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Uploaded bytes 45[8](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:9)5
Finished uploading artifact content to blob storage!
SHA256 hash of uploaded artifact zip is 5b8826632[9](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:10)943461363fd8ee25b751396a59e5ad597f1d4ee2ee982a1b3023cd
Finalizing artifact upload
Artifact test-logs.zip successfully finalized. Artifact ID 1806827904
Artifact test-logs has been successfully uploaded! Final size is 4585 bytes. Artifact ID is 1806827904
Artifact download URL: https://github.com/kubevirt/ipam-extensions/actions/runs/[10](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:11)370902776/artifacts/1806827904

@oshoval oshoval force-pushed the collect branch 3 times, most recently from 12ac56f to aeb29d4 Compare August 14, 2024 15:49
@kubevirt-bot kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 14, 2024
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 15, 2024

/hold cancel

the lane will protect from adding the error injection anyhow

@kubevirt-bot kubevirt-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 15, 2024
@EdDev
Copy link
Member

EdDev commented Aug 15, 2024

The files are uploaded in the job, will rework it

Run actions/upload-artifact@v4
With the provided path, there will be 6 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Uploaded bytes 45[8](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:9)5
Finished uploading artifact content to blob storage!
SHA256 hash of uploaded artifact zip is 5b8826632[9](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:10)943461363fd8ee25b751396a59e5ad597f1d4ee2ee982a1b3023cd
Finalizing artifact upload
Artifact test-logs.zip successfully finalized. Artifact ID 1806827904
Artifact test-logs has been successfully uploaded! Final size is 4585 bytes. Artifact ID is 1806827904
Artifact download URL: https://github.com/kubevirt/ipam-extensions/actions/runs/[10](https://github.com/kubevirt/ipam-extensions/actions/runs/10370902776/job/28709883078#step:10:11)370902776/artifacts/1806827904

Very nice.
Do you know what are the limitations of this store?

  • For how much time it will be available there?
  • Are there size limits?

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 15, 2024 •
edited
Loading

Very nice. Do you know what are the limitations of this store?

  • For how much time it will be available there?
  • Are there size limits?

Thanks, not sure, will post if i find

EDIT
Time limit - https://github.com/actions/upload-artifact
"Artifacts are retained for 90 days by default"

# Duration after which artifact will expire in days. 0 means using default retention.
# Minimum 1 day.
# Maximum 90 days unless changed from the repository settings page.
# Optional. Defaults to repository settings.
retention-days:

Size limit - https://github.com/actions/upload-artifact?tab=readme-ov-file#number-of-artifacts

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 15, 2024 •
edited
Loading

zip also available at the bottom of the summary page
https://github.com/kubevirt/ipam-extensions/actions/runs/10390828834?pr=57
it can also be deleted from there

@oshoval oshoval force-pushed the collect branch 2 times, most recently from 6c09ba4 to 170fa66 Compare August 18, 2024 07:42
@oshoval oshoval changed the title WIP e2e: Collect data in case of a failure e2e: Collect data in case of a failure Aug 18, 2024
@oshoval oshoval marked this pull request as ready for review August 18, 2024 07:43
@kubevirt-bot kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 18, 2024
@oshoval oshoval changed the title e2e: Collect data in case of a failure git actions, e2e: Collect git action data in case of a failure Aug 18, 2024
@oshoval oshoval changed the title git actions, e2e: Collect git action data in case of a failure e2e: Collect git action data in case of a failure Aug 18, 2024
@maiqueb
Copy link
Collaborator

maiqueb commented Aug 19, 2024

/approve

I'm assuming https://github.com/kubevirt/ipam-extensions/actions/runs/10438749437 to be an example of a run where the logs were captured.

Good work.

@kubevirt-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: maiqueb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 19, 2024
maiqueb
maiqueb approved these changes Aug 19, 2024
View reviewed changes
Copy link
Collaborator

@maiqueb maiqueb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

@kubevirt-bot kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 19, 2024
@kubevirt-bot kubevirt-bot merged commit 83a65d3 into kubevirt:main Aug 19, 2024
4 checks passed
@oshoval
Copy link
Collaborator Author

oshoval commented Aug 19, 2024 •
edited
Loading

I'm assuming https://github.com/kubevirt/ipam-extensions/actions/runs/10438749437 to be an example of a run where the logs were captured.

Indeed thanks
I will create a tag so we can consume it on CNAO (it will also consume your PRs fyi),
in the future we will have a way to consume test code without tags as you suggested once, but atm we dont yet.

@oshoval
Copy link
Collaborator Author

oshoval commented Aug 19, 2024 •
edited
Loading

Btw not related to this PR, but the XML is not generated, hence not collected
https://github.com/kubevirt/ipam-extensions/pull/57/files#diff-4af11422a4987e947e5a47adead7a30d32cdb2db82e2d3fe36f8e6cbe84d5ac5R74

@oshoval oshoval mentioned this pull request Nov 28, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maiqueb maiqueb maiqueb approved these changes

@qinqon qinqon Awaiting requested review from qinqon

Assignees

@maiqueb maiqueb

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@oshoval @EdDev @maiqueb @kubevirt-bot